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ABSTRACT 



Purpose : The aim of this study was to evaluate the effect of liquid crystal display (LCD) monitor type and observer 
experience on the diagnostic performance in soft-copy interpretations of maxillary sinus inflammatory lesions on 
panoramic radiographs. 

Materials and Methods : Ninety maxillary sinuses on panoramic images were grouped into negative and positive 
groups according to the presence of inflammatory lesions, using CT for confirmation. Monochrome and color LCDs 
were used. Six observers participated and ROC analysis was performed to evaluate the diagnostic performance. The 
reading time, fatigue score, and inter-/intra-observer agreements were assessed. 

Results : The interpretation of maxillary sinus inflammatory lesions was affected by the LCD monitor type used and 
by the experience of the observer. The reading time was not significantly different, however the fatigue score was 
significantly different between two LCD monitors. Inter-observer agreement was relatively good in experienced 
observers, while the intra-observer agreement for all observers was good with monochrome LCD but not with color 
LCD. 

Conclusion : The less experienced observers showed lowered diagnostic ability with a general color LCD. 
(Imaging Sci Dent 2011; 41 : 11-6) 
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Introduction 

In picture archiving and communication system (PACS) 
environments, PACS workstation is a vital tool for image 
interpretation because PACS equipments improve the effi- 
ciency, provide the image analysis tools, and enhance the 
image storage and distribution. 1 " 3 In PACS environment, 
computed radiography (CR) with photostimulable phosphor 
(PSP) plates is a suitable technique for extra-oral radiogra- 
phic examination 4 and has been applied to panoramic ra- 
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diography, 5 which is a useful and common examination 
for the oral and maxillofacial regions. 

Soft-copy interpretation on liquid crystal display (LCD) 
and cathode ray tube (CRT) monitors is as accurate as hard- 
copy interpretation and widely accepted in medical and 
dental practices. 6 " 12 If several factors including the monitor 
resolution, monitor luminance, image resolution, image 
bit depth, image receptor device, and ambient light are 
well regulated, the soft-copy interpretation is quite relia- 
ble. 13 " 16 

Dental implant therapy is an essential consideration for 
the rehabilitation of missing teeth. Implant surgery with 
maxillary sinus floor elevation has high implant survival 
rates. 1718 The assessment of anatomical structures and 
pathological lesions around the surgical site is one of the 
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requirements for successful dental implant therapy. 19 ' 20 
Therefore, diagnostic imaging is essential for dental implant 
therapy. 

Researchers assessed the value of panoramic radiography 
for the evaluation of the maxillary sinus. 21 ' 22 Those studies 
showed that a more experienced observer was likely to be 
more efficient with interpreting panoramic radiographs 
and suggested that specialists in oral and maxillofacial 
radiology were the best suited to evaluate the maxillary 
sinus on panoramic radiographs. 

The aim of this study was to evaluate the effect of LCD 
monitor type and observer's experience in soft-copy in- 
terpretation of maxillary sinus inflammatory lesions on 
panoramic radiograph. 

Materials and Methods 

Image acquisition and collection 

Fifty two CR panoramic radiographs were collected 
from patients (mean age 40.2 ± 17.3 years) who had taken 
CT (Somatom Sensation 10, Siemens AG, Forchheim, 
Germany) images on the same day. Panoramic images had 
been taken by experienced oral and maxillofacial radiogra- 
phers using two different panoramic radiography equip- 
ments (1, Orthopantomograph OP100, Instrumentarium 
Corp., Tuusula, Finland; 2, Cranex 3+ PAN, Orion Cor- 
poration Soredex, Helsinki, Finland). All CR images had 
been obtained using PSP image plates (12 x 10 inch) and 
read by an FCR system (Fuji Computed Radiography 
5000R, Fuji Photo Film Co. Ltd., Dusseldorf, Germany). 

The acquired digital raw data (2,010 x 1,670 pixels, 10 
bits, 6.7 pixel/mm) were sent to a PACS server and distri- 
buted to a PACS workstation. Then all images were down- 
loaded onto the local hard disk drive of the PACS worksta- 
tion as a DICOM (Digital Imaging and Communications 
in Medicine) format file before being read by observers. 
The average file size of each image was 3.66 MB, and the 



CR images were displayed in consecutive mode. 
Case selection 

A total of 90 maxillary sinuses from 52 patients were 
classified into negative or positive groups. The negative 
and positive classifications were confirmed on CT images 
by experienced oral and maxillofacial radiologist. Maxil- 
lary sinuses with surgical intervention and lesion involve- 
ment such as tumor, cyst and periapical infection, and 
congenital anomaly were excluded. On the basis of CT 
findings, the positive cases were defined as the presence 
of four types of inflammatory lesions on the sinus floor: 
1, thickened mucosa (n = 14); 2, polypoid mucosa («=23); 
3, air-fluid level (n=2); 4, totally mucosa-filled sinus (n= 
6). In this manner, 45 sinuses were classified as positive 
and the other 45 as negative. 

Study performance 

A high-resolution monochrome LCD (ME315L, Totoku 
Electric Co. Ltd., Tokyo, Japan) and a general color LCD 
(Nexviewl830, Dicon Co. Ltd., Seoul, Korea) monitors 
were applied as the CR image display methods in this 
study. The detailed specifications are given in Table 1 . To 
avoid the limitation of viewing angle with the LCD, the 
observer performed the image interpretation at a constant 
distance of 50 cm from the center of the monitor surface. 23 

All observers read the images twice on each monitor. 
This study consisted of four sessions as follows; the obser- 
vers interpreted 52 panoramic images on one monitor (ses- 
sion 1) and a few minutes later interpreted the images on 
the other monitor (session 2). The monitor for each session 
was randomly selected for each observer. All images were 
differently displayed on each monitor to minimize the 
memory effect between sessions 1 and 2. To avoid the 
learning bias, sessions 3 and 4 were performed over one 
week after sessions 1 and 2. 

All observers were proficient with PACS viewing soft- 



Table 1 . Specifications of two LCD monitors 





High resolution monochrome LCD 


Color LCD 


Diagonal size (inch) 


20.8 


18.1 


Spatial resolution (pixels) 


1536 x 2048 


1240 x 1024 


Maximum viewing angle (deg) 


+ 85 (horizontal) 


+ 80 (horizontal) 




± 85 (vertical) 


+ 80 (vertical) 


Contrast ratio 


600: 1 


300: 1 


Refresh rate (Hz) 


30 Hz 


60 Hz 


Gray scale (bits) 


10 


8 


Maximum brightness (cd/m 2 ) 


600 


200 
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ware (m-View-ps, Marotech, Seoul, Korea) as they used it 
in their daily practice. They were allowed to adjust the 
brightness and contrast of the images and also to magnify 
the images to their preference through a mouse control. 
No limitation was imposed on reading time. Ambient light 
was lowered as much as possible in order to eliminate 
reflections in the monitor. All observers were adapted 
fully to dark room conditions before the session was begun. 

The observers did not have any information of the pati- 
ents and proportion of positive cases. The definitions of 
the positive case were instructed to the observers, however 
any discussion for consensus about negative/positive cases 
was not allowed. The maxillary sinus above the molar 
region was determined as a region of interest (ROI). At 
each session, the absence or presence of inflammatory 
changes in the maxillary sinus was scored at each ROI 
using a five-point scale system: 1, definitely negative; 2, 
probably negative; 3, intermediate; 4, probably positive; 
5, definitely positive. The reading time was recorded for 
each observer during all sessions. After finishing each 
session, the observers were asked to determine the subjec- 
tive fatigue score using a ten-point scale from 1 (no feeling 
of fatigue) to 10 (extreme feeling of fatigue). 

Observers 

Six observers participated in this study. They were cate- 
gorized into three groups consisted of 2 observers: group 
A; faculty in oral and maxillofacial radiology with inter- 
preting experience of more than 15 years, group B; resident 
in oral and maxillofacial radiology with interpreting expe- 
rience of more than three years; group C, non-specialist 
with interpreting experience of less than two years. All 
observers had been using PACS software for over two 
years. 

Statistical analysis 

To evaluate the diagnostic performance, the receiver 
operating characteristic (ROC) analysis with calculation 
of the area under the ROC curve (A.) was performed using 
ROCKIT (0.9B beta version, Metz CE, Department of 
Radiology, The University of Chicago). Overall diagnostic 
performances according to the monitors and observers 
were determined by averaging the A z values. 

A statistical test was performed by using two-way analy- 
sis of variance (ANOVA) with interaction of the two fac- 
tors of LCD monitor type and observer experience, where 
A z was considered as replicates. Also, a statistical testing 
regarding the reading time and fatigue score was conducted 



using the Wilcoxon signed-rank test. Statistical analysis 
was performed using SPSS for Windows (version 12.0, 
SPSS Inc., Chicago, USA), and p<0.05 was considered 
statistically significant. 

To assess the decision agreement (inter- and intra-observ- 
er agreement), kappa (k) values were calculated such that 
k>0.75 was considered excellent agreement, 0.40 <k 
<0.75 was graded as fair or good agreement, and k<0.40 
was poor agreement. 

Results 

Table 2 shows the mean A z according to the monitors 
and observers and the overall diagnostic performances for 
the two monitors. Overall diagnostic performance of mono- 
chrome LCDs was superior to that of color LCDs. The 
diagnostic performance in experienced observers was 
higher than that in less experienced observers (Table 3). 
The A z value of each group decreased from group A to C 
serially. 

Two-way AVONA revealed that there was a significant 
difference between the LCD monitors and among the groups 
and also significant interactions between the monitor and 
group (p<0.05), which indicated that the diagnostic per- 
formance was dependent upon the LCD monitor type in 
soft-copy interpretation of the maxillary sinus (Table 4). 



Table 2. A- of observers 



Group 


Observer 


High resolution 
monochrome LCD 


Color LCD 






First 


Second 


First 


Second 


A 


1 


0.754 


0.802 


0.723 


0.732 




2 


0.749 


0.827 


0.810 


0.787 


B 


3 


0.762 


0.681 


0.672 


0.632 




4 


0.801 


0.775 


0.797 


0.781 


C 


5 


0.730 


0.800 


0.588 


0.627 




6 


0.723 


0.688 


0.577 


0.577 




Mean 


0.753 


0.762 


0.694 


0.689 



group A; oral and maxillofacial radiologists experienced more than 15 years, 
group B; oral and maxillofacial radiologists experienced more than three 
years, group C; non-specialist experienced of less than two years 



Table 3. Mean A z of each group at all sessions from ROC analysis 



Group 


Monochrome LCD 


Color LCD 


A 


0.783 


0.763 


B 


0.755 


0.720 


C 


0.735 


0.602 


Mean 


0.758 


0.692 



group A; oral and maxillofacial radiologists experienced more than 15 years, 
group B; oral and maxillofacial radiologists experienced more than three 
years, group C; non-specialist experienced of less than two years 
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Table 4. Results of two-way ANOVA with interaction of LCD 
monitor type and observer experience 



Table 6. Fatigue score of observers 



Factor 



p value 



Group (observer experience) 

AvsB 

AvsC 

B vs C 

LCD monitor type 

Interaction of LCD monitor type and 
observer experience 



< 0.001* 
0.002 f 

<0.001 t 
0.358 

< 0.001* 

0.015* 



♦Statistical significance (p<0.05), Statistical significance (p<0.05), group 
A; oral and maxillofacial radiologists experienced more than 15 years, group 
B; oral and maxillofacial radiologists experienced more than three years, 
group C; non-specialist experienced of less than two years 



Table 5. Total reading time (second) of observers at all session 



Observer 



Monochrome LCD 



Color LCD 



1 


1280.0 


1088.5 


2 


793.5 


685.5 


3 


776.0 


992.5 


4 


1061.0 


1104.5 


5 


832.0 


894.0 


6 


816.50 


905.5 


Mean 


941.0 


928.9 



Color LCD had a detrimental influence on the diagnostic 
performance of less experienced observers. In experienc- 
ed observers (group A), the diagnostic performance was 
comparable between the two LCD monitors. However, in 
less-experienced observers (group B, C), the diagnostic 
performance with the color LCD was lower than that with 
the monochrome LCD. 

When the reading time was averaged over all observers 
for the two monitors, there was no significant difference 
in the reading time (p> 0.05). However, there was a con- 
siderable individual variation in reading time ranging from 
685.5 to 1,280.0 seconds (Table 5). 

Table 6 reveals that the fatigue score ranged from 2 to 8 
for the two monitors. The mean fatigue score for the color 
LCD was significantly higher than that for the monochrome 
LCD (j?<0.05). All observers agreed that they felt more 
fatigued after using the color LCD. 

Tables 7 and 8 show the decision agreement. Inter-observ- 
er agreement of group A was relatively fair or good (0.40 
<k), however that of other groups was poor (k< 0.40). 
Kappa values for inter-observer agreement were higher 
for the specialists in oral and maxillofacial radiology. 
Intra-observer agreement for all groups was relatively fair 
or good, which indicated that all observers who participat- 
ed in this study had their own decision criteria except for 
one observer (observer 5) when using the color LCD. The 



Observer 

1 

2 
3 
4 
5 
6 

Mean* 



Monochrome LCD 

5 
2 
4 
3 
6 
5 

4.125 



Color LCD 

6 
7 



6.75 



♦Statistical significance (p< 0.05) 



Table 7. Inter-observer agreement of each group (kappa value) 



Group 


High resolution 
monochrome LCD 


Color LCD 


First 


Second 


First 


Second 


A 


0.390 


0.448 


0.397 


0.565 


B 


0.318 


0.209 


0.175 


0.297 


C 


0.005 


0.063 


0.029 


0.010 



group A; oral and maxillofacial radiologists experienced more than 15 years, 
group B; oral and maxillofacial radiologists experienced more than three 
years, group C; non-specialist experienced of less than two years 



Table 8. Intra-observer agreement of observers (kappa value) 



Observer 



High resolution 
monochrome LCD 



Color LCD 



1 


0.494 


0.262 


2 


0.687 


0.712 


3 


0.517 


0.557 


4 


0.555 


0.382 


5 


0.416 


-0.015 


6 


0.696 


0.659 



kappa values showed a tendency to be higher for the 
monochrome LCD than for the color LCD (Table 8). 

Discussion 

Inflammatory changes such as thickened mucosa on 
maxillary sinus floor are difficult to detect on plain radio- 
graphs. It is still essential to distinguish such changes from 
normal features. For example, dental implants should be 
inserted in bone with no pathological lesions. 

We selected panoramic radiography since it is the most 
useful radiological examination for jaw imaging. Lee et 
al reported that soft tissue lesions were difficult to detect 
by both specialists and non-specialists, however the speci- 
alists were more able to interpret lesions in the maxillary 
sinus on panoramic radiograph. 22 Our study confirmed 
that the experienced observers showed higher ability to 
detect lesions on the maxillary sinus floor. 
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There were several studies on the effect of monitor 
luminance for observer performance in soft-copy interpre- 
tation. 13 " 16 According to Herron et al, 24 hard-copy inter- 
pretation in chest radiography was not affected by lumin- 
ance above 260 cd/m 2 , which showed that the luminance 
below a threshold level had detrimental effects on the detec- 
tion performance in interpretation. This finding might be 
also applied to soft-copy interpretation. The observer per- 
formance in color LCD was inferior to that in monochrome 
LCD, which was, we thought, due to the monitor lumin- 
ance level. A decrease in luminance leads inevitably to a 
reduction in detection, which was similar to a study by 
Otto et al. 8 The maximum luminance of the color LCD in 
this study was only one-third of that of the monochrome 
LCD. This might resulted in decreased discrimination of 
low contrast tissue such as mucosal shadows in the max- 
illary sinus. As the light stimuli from the low contrast 
mucosa is very faint to the human visual system, the decr- 
eased contrast sensitivity seems to cause deterioration in 
the radiological interpretation process. 24 As Goo et al 7 
suggested that the observer performance was not affected 
under low ambient light, this study demonstrated that the 
detection performance might not be affected by an ambient 
light but was affected instead by a monitor luminance. 

In spite of the adjustments in window width and level, 
the less-experienced observers showed lower diagnostic 
abilities for detection of the maxillary sinus lesion when 
using the color LCD. This might be resulted from the 
observer's experience and level of radiological practice. 
Some investigators found that the level of experience influ- 
enced the observer performance in radiology. ' In 
case of experienced observers, the performance over vari- 
ous display methods (film, CRT, LCD etc.) were fairly 
even. This was in agreement with our results. 

The lack of difference in reading time between the two 
different monitors showed that the variation in reading 
time indicated the individual propensity to interpret an 
image. The reading time had no relation to the observer's 
experience. 

Otto et al 8 found that the diagnostic performance was 
improved with higher spatial resolution, which was consis- 
tent with our results. This indicated that the large number 
of pixels concomitant with small pixel size of a monochr- 
ome LCD would be suitable for detecting a fine distinction 
of mucosa. 

Goo et al 7 found that fatigue was higher at high levels 
of monitor luminance (100 fL) than low and middle levels 
(25, 50 fL), therefore fatigue in the monochrome LCD 
was expected to be higher than in the color LCD. However, 



our result was in opposition to their results. This might be 
related to the differences in reading time for each session 
in our study. The subjective determinations in the detection 
of low contrast tissue with monochrome LCD might make 
the observers less fatigued and also shortened the reading 
time in this study. 

Inter-observer agreement (kappa values) ranging from 
0.390 to 0.565 was fairly good only in experienced faculty 
of oral and maxillofacial radiology. Intra-observer agree- 
ment tended to be good with the monochrome LCD, and 
the reading time was not significantly different between 
the different monitors. Considering the A z values, fatigue 
score, and reading time, the high-resolution monochrome 
LCD monitor might be more suitable for primary interpre- 
tations in a PACS environment. However, because the 
monochrome monitors tend to increase the cost of PACS 
installation, a common color monitor can be considered 
as an alternative method. From our results, the observer's 
experience was an essential factor in soft-copy interpreta- 
tion. 

In conclusion, the less experienced observers in soft- 
copy interpretation of the maxillary sinus on panoramic 
radiographs showed lower diagnostic ability when using 
a general LCD monitor, therefore a high resolution mono- 
chrome LCD monitor might be more suitable for an image 
interpretation. 
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